


Guidelines 


4 We are a small group 
4 Interrupt the speaker 
4 Ask unrelated question 
4 Help us keep this fluid 


€ We can re-tool the agenda on-the-fly 
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Community Development 
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Topics 


# Community Status 
5 Why this is Important 
> Open Source Licensing 
2 Contributors and Working Groups 
5 Source Code 
+ Becoming a Developer 
=> Graph and Rolls 
2 Attributes 
5 Command Line 


€ Avoiding becoming a Developer 
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Community Status 
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WHY? 
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Rocks is almost 10 years old! 


4 Mission accomplished 
4 Rocks is the de facto open-source 
clustering solution 
4 Great user community 
> 2000+ on mailing list 
> Amazing signal to noise ratio 
4 Everything from 2 nodes cluster to top 10 
supercomputers 
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Hocks is almost 10 years old! 


+ 90% of development is 
2 NSF (and other grant) funded 
> Located at UC San Diego 


4 Need to diversify development 


2 More ideas, passion, and focus areas 
> More secure funding 
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OPEN SOURCE LICENSING 


O 2010 UC Regents 





Licensing / Copyrights 


http://www.rocksclusters.org/wordpress/?page_id=48 


4 Rocks is entirely open-source 


# BSD Attribution License 
> Standard UNIX open-source 
> Very friendly for derived works 


# We have not changed to the more recent non-attribution 
BSD license 


€ Copyrights are owned by University of California 
Regents 


€ 3" party code is a mix of licenses and copyrights 
> Most of Rocks bits are 3' party! 
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Attribution Clause 


This product includes software 
developed by the Rocks@ 
Cluster Group at the San Diego 
oupercomputer Center at the 
University of California, San 
Diego and its contributors. 
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Trademark 


invent@ucsd.edu 


4 The Rocks name and logo are registered 
trademarks. 


For fee licensing is available 
> Standard usage 


> Derivative usage (e.g. “ACME Rocks”) 
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Summary 

+ Rocks is open-source and free 

+ Use it any way you wish 

+ Make billions of dollars with it without even buying us a 


single beer 


€ Give us attribution 
€ License the name for commercial use 


€ These two things help keep us funded 
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EARLY COMMUNITY ROLLS 
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Sun Grid Engine 


4 Way back in 2004 
4 Rocks supported PBS 


4 Scalable Systems added SGE support 
> Laurence Liew, Najib Ninaba 
> 1º! external developers for Rocks 
5 Based in Singapore 


4 SGE Roll created from this 
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Torgue Roll 


# Way back in 2006 
4 SGE Roll was the favorite of core team 
4 Threatened to drop PBS Support 


4 The Computer Center, University of Tromsg 
5 Roy Dragseth 
> 2™ major external developer 
5 Based in Norway 


4 Continues to develop and support Torque Roll 
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We were “big” overseas 
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First major attempt for actively recruit developers 


WORKING GROUPS 
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Working Groups 


4 Purpose 
> Fill gaps from core development team 
> Handle issues off the core road map 
> Make Rocks a more flexible solution 
4 Success Metrics 
> Number of Rolls produced 
> Amount of new Documentation (Wiki, ...) 
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Software Update WG 


https://wiki.rocksclusters.org/wiki/index.php/Software Update (SUWG) 


4 Started early 2008 
> Threads on yum updates increased 
2 Core team said "don't do it” 
5 Advocates said "it works for me" 
5 WG was recruited to address the issue 


$ Best practices defined 
2 Exclude lists 
2 Additional docs on custom restore Rolls 
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Status 


# Community Interest: High 
4 Documentation: Moderate 
4 Rolls Produced: None 


+ Summary: Some real interest but needs 
leadership. 
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Thumper Working Group 


https://wiki.rocksclusters.org/wiki/index.php/Rocks on Thumper 


+ Began with Rocks Solaris port 
5 Sun funded 
5 How to Manage ZFS NAS appliances 
> Specifically Sun Thumper 


4 Core team lead effort 
€ Used by several groups at UCSD 
# Software is released 
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Status 


# Community Interest: Low 
4 Documentation: Good 
4 Rolls Produced: Jumpstart 


+ Summary: Excellent activity with a small 
UCSD audience. Needs to build a larger 
user base. 
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Rolls Working Group 
https://wiki.rocksclusters.org/wiki/index.php/Rolls Working Group 
4 Started early 2009 


> Developing free versions of commercial Rolls 
> Organized by Stanford University 


4 Self-organized group of a 3 individuals 
4 Good initial offering of Rolls 
e Struggled with mailing list support 


O 2010 UC Regents 23 





Status 


# Community Interest: High 
4 Documentation: Average 
4 Rolls Produced: Good 


+ Summary: Excellent start, needs help with 


user support and keeping current with 
Rocks releases. 
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Triton Working Group 
http://tritonresource.sdsc.edu/ 
4 Started 2009 


> Developing Roll for large production cluster 
> Every piece of SW on system is part of a Roll 
> Includes commercial software 


+ Amazing set of Rolls (20+) to be released 
4 Triton group is here at SDSC 
+ No organized presence on Rocks list 
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Status 


4 Community Interest: Good 
+ Documentation: Good 
4 Rolls Produced: Excellent 


+ Summary: Highly productive group, but 
meets weekly with member(s) of Rocks 
core team. Phil is also their boss. 
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Great idea, some good traction, but not what we want 


OVERALL GRADE: C- 
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What can UCSD do better? 


€ WG phone/video conference 
5 WG to Core team 
5 WG All hands 
€ Need to communicate roadmaps between WGs and 
Core team 
2 Ease release tracking 
5 No surprises (e.g. Rocks Command Line) 
+ Where should support issues go? 
5 Main list 
5 A new WG list 
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Ideas? 


4 Docs in dev process 
5 Devel guide out of date 
5 Mine mailing list for solution 
4 Developer Cloning Process 
2 Jumpstart guide to development 
+ RESOLVED tag on mailing list 
+ Bug/Issue searchable database 
5 RH is a good example of this 
+ IRC 6 2010 UC Regents 29 
4 AIM Address Book (non-indexed) 





What can you do better 


e Tell us what you want 
5 Complain 
>A lot 
5 But, nicely 
€ Ask for help to start a new working group 
€ Join an existing working group 
4 We are starting this today 


@ 2010 UC Regents 30 





SOURCE CODE 


O 2010 UC Regents 





Version Control 


4 Core team uses CVS 
=> That's what we started with 
> We aren't changing anytime soon 
4 CVS access available to very few people 
5 [oo risky 
> Access control is a pain 
> Release management difficult 
# We use Mecurial for all non-core development 
+ Mecurial synced to CVS every 10 minutes! 
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Workflow 


fyp.rocksclusters.org 


-— 





cvs up 





O 2010 UC Regents 33 





e 


Example 


$ hg clone http://fyp.rocksclusters.org/hg/rocks-5.3 
destination directory: rocks-5.3 

real URL is http://fyp.rocksclusters.org/hg/ 
rocks-5.3/ 

requesting all changes 

adding changesets 

adding manifests 

adding file changes 

added 1 changesets with 2815 changes to 2815 files 
2815 files updated, 0 files merged, 0 files removed, 
0 files unresolved 
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Issues 


e Mecurial is slow 

4 Transaction based 
> Any aborted operation rolls back 
2 Do not stop the clone 


€ Patch sets can be tedious 
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Advantages - Freedom 


4 Publish your own repository 
4 No need to even commit back to core 
+ Commit broke code and only hurt yourself 


€ Core Rocks remains stable 
€ HG clones innovate 
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Notes 


4 For code older than 5.0 


o ftp://ftp.rocksclusters.org/pub/rocks/rocks-src 
2 rocks-2.3 to rocks-4.3 


e We are not tied to this workflow 
e We are not tied to HG 


+ Other workflow suggestions are solicited 
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Becoming a Developer 





Resume (9 11:20 
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The Rocks engine 


GRAPH AND ROLLS 
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Rocks Configuration Graph 


094 D 
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The XML Graph Includes 


4 Nodes 
5 Single purpose modules 
2 Kickstart file snippets (XML tags map to kickstart commands) 
2 Approximately 200 node files in Rocks 
e Graph 
> Defines interconnections for nodes 
> Think OOP or dependencies (class, include) 
2 Asingle default graph file in Rocks 
e Macros 
5 SQL Database holds site and node specific state 
5 Node files may contain &state; entities (attributes) 
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Composition 


+ Aggregate Functionality 


4 scripting lsA 
2 perl-development 
2 python-development 
2 tcl-development 





python- 
development 


perl- tel- 
development development 
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Traverse by Attributes 


€ ifx11 == TRUE 
> client ISA x11 


€ if rsh == FALSE 
2 client lsNotA rsh 


+ Most important slide in 
this session 





€ RCL allows you to control 
the graph 
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Think of this as Cluster DNA 
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Iu 
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HOLL FUNDAMENTALS 
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Cluster Software | 
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Rolls Break Apart Rocks 








Applications 


Middleware 


Kernel 





Rolls: Modifying a Standard System Installer to Support User-Customizable Cluster Frontend Appliances. Greg Bruno, Mason J. Katz, 
Federico D. Sacerdoti, and Phil M. Papadopoulos. /EEE International Conference on Cluster Computing, San Diego, California, Sep. 2004. 
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Our Graph Has Colors 
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Rolls are sub-graphs 


€ A graph makes it easy to 'splice' in new nodes 


4 Each Roll contains its own nodes and splices 
them into the system graph file 


graphs nodes Makefile src 


default sge-client.xml ganglia globus sge use 
sge-globus.xml 
sge-server.xml 


sge.xml src code src code src code 


sge.xml 
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STARTING FROM THE EMPTY 
SET 
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. Simplified Example 


(base, hpc, sge, bio} 
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Two different Clusters 
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ATTRIBUTES 
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Attributes 


4 Attributes can be set at 4 levels: 
=> Globally 
* ‘rocks set attr 
=> By appliance type 
* 'rocks set appliance attr' 
5 By OS (linux or sunos) 
* ‘rocks set os attr’ 
=> By host 
* “rocks set host attr’ 
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Attributes 


4 Example, set the public IP address of a remote 
frontend that is used during a ‘central’ 
installation: 


# rocks set host attr vi-l.rocksclusters.org ١ 
Kickstart PublicAddress 137.110.119.118 
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e Attributes 


t rocks list host attr tile-0-0 


HOST 

tile-0- 
tile-0- 
tile-0- 
tile-0- 
tile-0- 
tile-0- 
tile-0- 
tile-0- 
tile-0- 
tile-0- 
tile-0- 
tile-0- 
tile-0- 
tile-0- 
tile-0- 
tile-0- 
tile-0- 
tile-0- 
tile-0- 
tile-0- 
tile-0- 
tile-0- 
tile-0- 
tile-0- 
tile-0- 
tile-0- 
tile-0- 
tile-0- 


0 
0 
0 
0 
0 
0 
0 
0 
0 
0 
0 
0 
0 
0: 
0: 
0 
0 
0 
0 
0 
0 
0 
0 
0 
0 
0 
0 
0 


ATTR 

Info CertificateCountry 
Info CertificateLocality 
Info CertificateOrganization 
Kickstart DistroDir 
Kickstart PrivateAddress 
Kickstart PrivateBroadcast 
Kickstart PrivateDNSDomain 
Kickstart PrivateDNSServers 
Kickstart PrivateGateway 
Kickstart PublicDNSServers 
Kickstart PublicGateway 
Kickstart PublicHostname 
Kickstart PublicKickstartHost 
Kickstart PublicNTPHost 
Kickstart PublicNetmask 
Kickstart PublicNetmaskCIDR 
Kickstart PublicNetwork 
Kickstart Timezone 

Server Partitioning 

arch 

hostname 

rack 

rank 

rocks version 

HideBezels 

HttpConf 

HttpConfigDirExt 

HttpRoot 


VALUE 

US 

San Diego 

CalIT2 

/export/rocks 

10.1.1.1 

10.1.255.255 

local 

10.1.1.1 

10.1.1.1 

132.239.0.252 
137.110.119.1 
vizagra.rocksclusters.org 
central.rocksclusters.org 
pool.ntp.org 
255.255.255.0 

24 

137.110.119.0 

America/Los Angeles 
force-default-root-disk-only 
x86 64 

tile-0-0 

0 

0 

5.2 

false 

/etc/httpd/conf 
/etc/httpd/conf.d 

/ var /www/html 
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Edge Conditionals 


4 Use attributes to conditionally traverse 
edges of the configuration graph 


«edge from="client" cond="rsh"> 
<to>rsh</to> 
</edge> 


+ If ‘rsh’ evaluates to ‘true’, then the edge 
from ‘client’ to ‘rsh’ will be traversed 
> Default value is ‘false’ 
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Edge Conditionals 


4 To set a conditional attribute: 


tt rocks set attr rsh true 


+ Edge conditionals are attributes 
# Can also be set at 4 levels: 

=> Globally 

=> By appliance type 

5 By OS (linux or sunos) 

=> By host 


O 2009 UC Regents 


62 





COMMAND LINE 
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Evil Commands 


Usage: add-extra-nic [-hvv] [-p password] [-u host] [-d database] [--help] 
[--list-rcfiles] [--list-project-info] [--verbose] [--dump] [--del] [--list] 
[--verbose] [--no-update] [--no-modify] [--dryrun] [--rcfile arg] [--host host] 
[--password password] [--db database] [--user host] 

[--if interface (default: ethl)] [--mac mac address] 

[--module linux driver module name] [--ip ip address] 

[--netmask netmask (default /24)] [--gateway ip address of gateway] 
[--name hostname on new interface] [--site client ip] node 

Usage: rocks-dist [-hvcpv] [-p password] [-u host] [-d database] [-a arch] 

[-d dirname] [-g path] [-1 lang] [-r release] [--help] [--list-rcfiles] 
[--list-project-info] [--verbose] [--copy] [--debug] [--graph-draw-invis-edges] 
[--graph-draw-order] [--graph-draw-edges] [--graph-draw-key] [--graph-draw-all] 
[--graph-draw-landscape] [--install] [--verbose] [--with-rolls-only] [--clean] 
[--notorrent] [--rcfile arg] [--host host] [--password password] 

[--db database] [--user host] [--arch architecture] [--comps path] 

[--dist dirname] [--graph-draw-size arg] [--graph-draw-format arg] 
[--mirror-dir dirname] [--mirror-host hostname] [--root dirname] 

[--cdrom /mnt/cdrom] [--with-roll rollname-rollversion] 


[--path single path item] command 
Available commands: 
dist dvd makecontrib makesitenodes copycd usb copyroll cdrom paths graph dist2mirror 
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Command Line as API 


4 Lack of consistency in Rocks commands 
5 add-extra-nic (15 flags) 
© 411put 
5 rocks-dist 
> dbreport (~ a dozen reports) 
# Not extensible to other groups 
5 How do | add a flag to an existing command? 
> How do | add a new command? 
> How do | document my command? 
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DO Over 


e Consistent 

5 Interface 

> Argument parsing 

> Usage / Help 

Extensible 

o Easy to add commands (3" party rolls) 

> Easy to modify commands 

Easy to guess the right command 

Purge all —flags from Rocks 

Hide the SQL database (and underlying schema) 


Inspired by Trac 


* 


© + 99 
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Verb Based 


+ “add”, “set”, “enable”, ... 


5 Modify the cluster 
database 


€ “list”, "dump", "report" 
2 Inspect the cluster 
database 


€ About 20 verbs in the 
command line so far 


€ You can even add your 
own 
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Grammar 


4 rocks <verb> <object...> <subject> <params.. 


+ Object is general to specific 
> “host” "interface" 
5 "network" "subnet" 
5 VIZ" "layout 
$ Subject is typed 
5 host 
> appliance 
2 network 
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Implementation 


4 Python 
5 Similar to existing dbreport code 
> Very small modules 
€ Command line is identical to the directory 
hierarchy 
> Verbs are directories 
> Objects are directories 
> Subjects are init .py files 
+ Commands are added by adding directories 
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distribution interface init DN 
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add host pxeaction init .py 


rocks add host pxeaction 
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di 
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T. 
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create mirror . init__.py 
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Rolls Can Add Commands 


À 


















































e Similar to the "nimii 
configuration graph 5 2 3 
+ Rolls can add TE + 
command line HEN 
> Files : commands w— a z 
> Directories : verbs and a Ez 
objects as à 

+ Think hard before “Lg 
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add 


4 Creates new entries 
in the cluster 















database 

4 Examples: 
2 Hosts TES. 
> Appliances CN 
=> Rolls S 


interface dni | py 
a E: 
network init By 


i 72 


— init py 





. init .py 








WOON 65 Un P خم رح رن‎ 


rocks add distribution 


import rocks.comnands 


class Command(rocks . commands . Di stributionArgumentProcessor, 


rocks . commands . add. command) : 


Add a distribution specification to the database. 


«arg type-'string' name-" distribution": 
Name of the new distribution. 
</arg> 


«example and="add distribution rocks-dist'> 
Adds the distribution named "rocks-dist" into the database. 
</example> 


def run(self, params, args): 
if len(args) != 1: 
self.abort('must supply one distribution) 
dist = args[0] 


if dist in self.getDistributionNamesQ): 
self.abort('distribution "965" exists" 96 dist) 


self.db.execute("" "insert into distributions (name) values 
('Xs')""" 96 dist) 
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dump 


4 Returns cluster 
database information 
in the form of rocks 
command lines 

4 Examples: 

> Hosts 
2 Network 


$ Same as —dump flag 
on insert-ethers 
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rocks dump host 


# rocks dump host 

/opt/rocks/bin/rocks add host vizagra cpus=1 rack=0 rank=0 membership="Frontend" 
/opt/rocks/bin/rocks add host tile-0-1 cpus-2 rack-0 rank=1 membership="Tile" 
/opt/rocks/bin/rocks add host tile-0-0 cpus-2 rack-0 rank-0 membership="Tile" 
/opt/rocks/bin/rocks add host tile-0-2 cpus-2 rack-0 rank=2 membership="Tile" 
/opt/rocks/bin/rocks add host tile-0-3 cpus-2 rack-0 rank=3 membership="Tile" 
/opt/rocks/bin/rocks add host tile-1-3 cpus-2 rack=1 rank=3 membership="Tile" 
/opt/rocks/bin/rocks add host tile-1-2 cpus-2 rack=1 rank=2 membership="Tile" 
/opt/rocks/bin/rocks add host tile-1-1 cpus-2 rack=1 rank=1 membership="Tile" 
/opt/rocks/bin/rocks add host tile-1-0 cpus-2 rack=1 rank-0 membership="Tile" 
/opt/rocks/bin/rocks add host tile-2-0 cpus-2 rack-2 rank-0 membership="Tile" 
/opt/rocks/bin/rocks add host tile-2-1 cpus-2 rack-2 rank=1 membership="Tile" 
/opt/rocks/bin/rocks add host tile-2-2 cpus=2 rack-2 rank=2 membership="Tile" 
/opt/rocks/bin/rocks add host tile-2-3 cpus-2 rack-2 rank=3 membership="Tile" 
/opt/rocks/bin/rocks add host tile-3-0 cpus-2 rack=3 rank-0 membership="Tile" 
/opt/rocks/bin/rocks add host tile-3-1 cpus-2 rack=3 rank=1 membership="Tile" 
/opt/rocks/bin/rocks add host tile-3-2 cpus-2 rack-3 rank=2 membership="Tile" 
/opt/rocks/bin/rocks add host tile-3-3 cpus=2 rack=3 rank=3 membership="Tile" 
/opt/rocks/bin/rocks add host tile-4-0 cpus-2 rack-4 rank-0 membership="Tile" 
/opt/rocks/bin/rocks add host tile-4-1 cpus-2 rack-4 rank-1 membership="Tile" 
/opt/rocks/bin/rocks add host tile-4-2 cpus=2 rack-4 rank-2 membership="Tile" 
/opt/rocks/bin/rocks add host tile-4-3 cpus=2 rack-4 rank=3 membership="Tile" 
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ait 


command . init .py 





rocks list host 


# rocks list host 
HOST MEMBERSHIP CPUS RACK RANK COMMENT 
vizagra: Frontend 1 0 H ------- 
tile-0-1: Tile 
tile-0-0: Tile 
tile-0-2: Tile 
tile-0-3: Tile 
tile-1-3: Tile 
tile-1-2: Tile 
tile-1-1: Tile 
tile-1-0: Tile 
tile-2-0: Tile 
tile-2-1: Tile 
tile-2-2: Tile 
tile-2-3: Tile 
tile-3-0: Tile 
tile-3-1: Tile 
tile-3-2: Tile 
tile-3-3: Tile 
tile-4-0: Tile 
tile-4-1: Tile 
tile-4-2: Tile 
tile-4-3: Tile 


NN NN NN NNN NN NNN رخ رخ رخ ارخ رخ‎ NN 
SS E À LU 00 جم رح رح رح رح رن‎ KA KA POS © © 
LA خم رح‎ OU خم رح‎ © LPM كم © © خم‎ N LM A P © Hm 
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sel 


+ Modifies entries in the 
cluster database 


+ Examples: 
5 Network Interfaces 
5 Appliance Assignment 
=> Rack / Rank 


€ add-extra-nic 
2 Rocks add host interface 
2 Rocks set host interface 
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# 


. init .py 


network 








start int DN 
= É 
stop init py 







config 


sync 
















bootflags 





comment 





interface module 






membership subnet 






pxeboot 











Lus py 
netmask Ix DN 
SCH? B: DN 
KA 
“if, 78 


plugin_dns.py 





e 


start / stop 


+ Start and stop 
something 


+ NULL commands 


€ Reserve the verbs for 
use on other Rolls 

€ Think "abstract base 
class" 
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Sync 


+ Synchronizes the 
database state to 
software 
configuration files 

4 Similar to the old 


"Insert-ethers — 
update” 
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Extensibility 


# New commands 
5 Add directories 
2 Add init .py code 
4 Existing commands 
> Some commands can be extended 
> Plugins 
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rocks sync users 


4 Run after useradd 
2 Populate auto.home 
> Cleanup password file 
5 Send 411 files 


€ Two plugins = 
> Fixnewusers GC, = d 
2 411 sync users plugin 411.py 
e Partial Ordering P 


plugin. fixnewusers.py 


€ Other Rolls can add more 
plugins to this command 


+ Command must be design for 
plugins (not default) 
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Int py 


LO OP ل-‎ 65 Ln BR ND جم‎ 


import rocks. commands 


class Command(rocks . commands . sync. command): 
Update all user-related files (e.g., /etc/passwd, /etc/shadow, etc.) 
on all known hosts. Also, restart autofs on all known hosts. 


«example amd-'sync users'> 
Send all user info to all known hosts. 
«/example- 


def run(self, params, args): 
self.runPluginsO) 
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411 plugin 


O oo ND لح آنل‎ LU رح‎ KA 


import os 
import rocks. commands 


class Plugin(rocks. commands . Plugin) : 
"""Force a 411 update and re-load autofs on all nodes'"" 


def provides(self): 
return "411" 


def requires(self): 
return ['fixnewusers' ] 


def run(self, args): 
# 


# force the rebuild of all files under 411's control 
# 


for line in os.popenC'make -C /var/411 force').readlinesQ: 


self.owner.addText(line) 


# 

# restart autofs on all knom hosts 

# 

and = '/opt/rocks/bin/tentakel "service autofs reload" 

for line in os.popen(and). readlines(): 
self.owner.addText(line) 
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auto.home / passwd plugin 


import os 
import string 
import rocks. commands 


PR 


Ex eS edi 


LA 


a 








PARENÊSOSNDUBUNG 


LA 
“q 


class Plugin(rocks. commands . PLugin): 


"""Relocates home directories to /export and fixes autofs.homé 


def provides(self): 
return 'fixnewusers' 


def run(self, args): 
# scan the password file for any '/export/home' entries 
# this is the default entry as setup by useradd 
new users - [] 
default dir = '/export/home/" 


file = open('/etc/passwd', Tri" 


for line in file.readlinesQ): 
1 = string.splitCline[:-1], ":") 


if len(1) < 6: 
continue 


username = 1[0] 
homedir = 1[5] 


if homedir[:len(default dir)] = default dir: 
new users .append(usernane) 
file.close() 


hostname = '%s.%s' 96 N 
Cself.db.getGlobalVar('Kickstart', 'PrivateHostname'), 
self.db.getGlobalVar('Kickstart', 'PrivateDNSDomain')) 


for user in new users: 


# for each new user, change their default directory to 
# /home/<username> 


end = */usr/sbirvusermod -d 482008 UC Regents 


(os.path. join('/home", user), user) 
for line in os.popen(amd) . readlinesO: 
self.owner.addText(line) 


# then update the auto.home file 
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Argument Processing 


4 rocks <verb> <object...> <subject> 
<params...> 


4 Subject is typed by first object 
2 host -> one or more hostname 
2 roll -> one or more roll names 


4 Params are in key=value form 
+ Same as —flag=value but easier to read 
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e 


Helper classes and functions 


+ ArgumentProcessors 
> Class to parse the subject in a standard way 
5 Exists for hosts, rolls, appliances, ... 
+ Parameters Parsing 
> fillPositionalArgs 
> fillParams 
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HostArgumentProcessor 


+ Command must derive from 
rocks . commands . HostArgumentProcessor 


+ self. getHostnames(args) 
2 Return a list of hostname as they appear in the cluster 
database 
> If args = None all the host in the cluster are returned 
2 args can also be a group 
e RackO, rack1 
5 Or an appliance type 
* Compute, Tile, ... 
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1: import rocks.commands 

2: 

3: class command(rocks. commands. HostArgumentProcessor , 

4: rocks . commands .1ist.comnand) : 

5: pass 

6: 

7: class Command(command): 

8: in 

9: List the membership, CPU count, physical position info and comment for 
10: a list of hosts. 

11: 

12: «arg optional-'1' type-" string" name-'host' repeat='1'> 
13; Zero, one or more host names. If no host names are supplied, info about 
14: all the known hosts is listed. 

15: </arg> 

16: 

17: «example and='list host compute-0-0'> 

18: List info for compute-0-0. 

19: </example> 

20: 

21: «example amd-'list host": 

22: List info for all known hosts. 

23: </example> 

24: CH 

25: 

26: def run(self, params, args): 

27: self.beginOutputO 

28: 

29: for host in self.getHostnames(args): 

30: self.db.execute("""select m.name, n.cpus, 
31: n.rack, n.rank, n.comment from 
32: nodes n, memberships m where 

33: n.membership=m.id and n.name="%s"""" 96 host) 
34: self.addOutput(host, self.db.fetchoneQ) 
35: 

36: self.endOutput(header-['host', "membership" , 

37: "cpus", "rack", "rank", "comment" 1 

38: 
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args = None 


# rocks list host 
HOST MEMBERSHIP CPUS RACK RANK COMMENT 


vizagra: Frontend 1 0 Q £------- 
tile-0-1: Tile À Q 1  ------- 
tile-0-0: Tile 2 0 H £------- 
tile-0-2: Tile 2 0 2 ------- 
tile-0-3: Tile 2 0 3 ------- 
tile-1-3: Tile 2 1 381112233533 
tile-1-2: Tile 2 1 2 — ------- 
tile-1-1: Tile 2 1 1 ------- 
tile-1-0: Tile 2 1 HEEL 
tile-2-0: Tile 2 2 ]اام‎ 1111131 
tile 21: Tile € 2008 UÊ Regent? (١ ١ 1| 0 
tile-2-2: Tile 2 PREREEREEERE 
tile-7-3º Tile 7 E Do OOS 





args = list of hosts 


# rocks list host tile-0-0 10.255.255.253 tile-3-Ø. local 


HOST MEMBERSHIP CPUS RACK RANK COMMENT 
tile-0-0: Tile 2 0 Q £------- 
tile-0-1: Tile 2 0 1  ------- 
tile-3-0: Tile 2 3 Q £------- 
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args = rack 


# rocks list host rack2 


HOST MEMBERSHIP CPUS RACK RANK COMMENT 
tile-2-0: Tile 2 2 6 ------- 
tile-2-1: Tile 2 2 1 00 
tile-2-2: Tile 2 2 2 — ------- 
tile-2-3: Tile 2 2 3 — ------- 
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args = appliance type 


# rocks list host tile 

HOST MEMBERSHIP CPUS RACK RANK COMMENT 
tile-0-0: Tile 2 0 H #£------- 
tile-0-1: Tile 
tile-0-2: Tile 
tile-0-3: Tile 
tile-1-0: Tile 
tile-1-1: Tile 


NNN N n n 


A KA KA KA © © © 


tile-1-3: Tile 2 
tile-2-0: Tile © 2008 UC Regent? 
tile-2-1: Tile 2 2 


l KA OWNF OWN F 





Any combination is fine 


# rocks list host tile-2-0 rack1 frontend 


HOST 


tile-1-0: 
tile-1-1: 
tile-1-2: 
tile-1-3: 
tile-2-0: 


vizagra: 


MEMBERSHIP CPUS RACK RANK COMMENT 


Tile 
Tile 
Tile 
Tile 
Tile 
Frontend 


2 1 0 
2 1 1 
2 1 2 
2 1 E 
2 2 0 
1 0 0 
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e 


ArgumentProcessors 


Class Name 
ApplianceArgumentProcessor 
DistributionArgumentProcessor 
HostArgumentProcessors 
MembershipArgumentProcessor 
NetworkArgumentProcessor 
RollArgumentProcessor 


Helper Function 
getApplianceNames 
getDistributionNames 
getHostnames 
getMembershipNames 
getNetworkNames 
getRollNames 
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e RollArgumentProcessor 





1: import os 

2: import stat 

3: import time 

4: import sys 

5: import string 

6: import rocks. commands 

7: 

8: 

9: class Comand(rocks. commands . Rol lArgumentProcessor, 

10: rocks . commands .List . command): 

11: SCH 

12: List the status of available rolls. 

13: 

14: «arg optional="1' type-'string' name-" roll" repeat='1'> 

15: List of rolls. This should be the roll base name (e.g., base, hpc, 
6: kernel). If no rolls are listed, then status for all the rolls are 
17: listed. 

18: «/arg» 

19: 

20: «example and="list roll kernel'> 

21: List the status of the kernel roll 

22: </example> 

23: 

24: «example and="list roll": 

25: List the status of all the available rolls 

26: </example> 

27: KA 

28: 

29: def run(self, params, args): 

30: 

31: self .begindutputQO 

32: for (roll, version) in self.getRollNames(args, params): 
33: self.db.execute("""select version, arch, enabled from 
34: rolls where name='%s' and version="%s"""" 96 
35; (roll, version)) 

36: for row in self.db.fetchallO: 

37: self.addutput(roll, row) 

38: 

39: self.endOutput(header-['name', "version", 'arch', 'enabled'], 
40: trinOwner-Q) 
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No Parameter 


# rocks list roll 


NAME 
VIZ: 
sge: 
kernel: 
updates: 
java: 
xen: 
CentOS: 
ganglia: 


web-server: 


base: 


VERSION ARCH ENABLED 


| Ul UT UI UI Ui B UI UI UI ul 
| C © © خم‎ © A خم‎ © © 


.0 


1386 yes 
1386 yes 
1386 yes 
1386 yes 
1386 yes 
1386 yes 
1386 yes 
1386 yes 
@ 200b BOB NE S 
1386 yes 


e, On gan 
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Version Parameter 


# rocks list roll versionz4.3.2 
NAME VERSION ARCH ENABLED 
java: 4.3.2 1386 yes 
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Summary 


4 ArgumentProcessors standardize the 
handling of command line subjects 


4 Calling the helper function with an empty 
list returns all subject in the database 


4 HostArgumentProcessor knows about 
more than just host names 


4 RollArgumentProcessor can filter on 
versions 
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fillParams 


4 Create local variables based on 
command parameters (key=value) 


+ Argument a list of (key, default) tuples 


4 If the parameter is not found on the 
command line the default value is used 
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AN Heh. ei سبي‎ eei. A MR dU NEM Fb "Ser Ae St ael He. I Tha Si Sach Za 


</example> 


def maketorrent(self, filename, data): 
info = {} 
info['length'] = os.stat(filename) [stat ST SIZE] 
info['name"] = os.path.basenane(fi Lename) 


info'] = info 


rocks Create.torre nt 


file = open('%s.torrent' % (filename), 
file.write(encoded) 
file.closeQ 


def run(self, params, args): 
if len(args) != 1: 
self.abort('must supply one file’) 
filename = args[?] 


(timestamp, ) = self.fillParams([C timestamp', time.timeQ)]) 


try: 

creation date = int(timestamp) 
except: 

creation date = int(time.time()) 
data = (1 
# 
# announce string 
# 


localhost = self.db.getGlobalVar('Kickstart', 'PrivateAddress') 
data['announce'] = 'http://96s:7625/announce' 96 (localhost) 


data['creation date'] = creation date 
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basename, rack, rank = host.split( -') 
self.db.execute("" "select m.name from 

appliances a, memberships m where 

a.name="%s" and m.appliance-a.id'"" 96 basename) 
membership, = self.db.fetchone(O) 
rack = int(rack) 


rocks add host 


rack = None 
rank = None 


# fillParams with the above default values 


(membership, numCPUs, rack, rank) = self.fillParamsQ 
[C'membership", membership), 
C'epus' , 1), 
C'rack', rack), 
Crank", rank)]) 


if not membership: 

self.abort('membership not specified) 
if rack — None: 

self.abort('rack not specified") 
if rank — None: 

self.abort('rank not specified') 


self.db.execute("" "insert into nodes 
(site, name, membership, cpus, rack, rank) 
values 


WAN 1 

(select id from memberships where name-'Xs'), 

Wich 5 

'%d' f 

'%d Wen % 

(host, membership, int(nu 5de WIS Arg er páCnankO)) 
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fillPositionalArgs 


+ Allows for parameters to have implied keys (just values 
on command line) 


4 This is an optimization for ease of use, not ease of 
software 
€ Argument is a list of keys 


2 No default value processing, if a key is specified it is required 
2 Use this only when a parameter is required 
+ Example: 


# rocks set network netmask optiputer netmask=255.255.255.0 
# rocks set network netmask optiputer 255.255.0.0 
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</arg> 


«arg type-'string' name-'netmask'» 
Netmask that named networks should have. 
«/arg» 


«param type-" string" name-'netmask'» 
Can be used in place of netmask argument. 
</param> 


rocks set.network netmask 


Sets the netmask for the "optiputer" network to a class-c address 


space. 
</example> 


«example amd-'set network netmask optiputer netmask=255.255.255.0'> 
Same as above. 
</example> 


«example amd-'set network netmask optiputer cavewave 255.255.0.0'> 
Sets the netmask for the "optiputer" and "cavewave" networks to 

a class-b address space. 

</example> 


<related-add network</related> 
<related>set network subnet</related> 


def run(self, params, args): 
(args, netmask) = self.fillPositionalArgs(C netmask' ,)) 


if not len(args): 

self.abort('must supply network') 
if not netmask: 

self.abort('must supply netmask") 


for network in self.getNetworkNames(args): 


self .db.executeC"""update subnets set netmaske'Xs' where 
subnets .nane="%s'""" 96 (netmask, network) 
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rocks Sel 


Sets the MAC Address for the eti, device on host compute-0-0. 
</example> 


«example and='set host interface mac compute-0-0 iface=eth1 mac-00:0e:0c:a7:5d:ff'» 
Same as above. 
</example> 


«example amd-'set host interface mac compute-0-0 iface-ethl mac=NULL '< 
clears the mac address from,the ho 


<!-- cross refs do not exist yet ost | nte rface 


<related>set host interface iface</related> 
<related>set host interface ip</related> 
<related>set host interface gateway</related> 
<related>set host interface module</related> 
==> 


<related>add host</related> 


def run(self, params, args): 
(args, iface, mac) = self.fillPositionalArgs(( iface', 'mac')) 
hosts = self.getHostnames(args) 


if len(hosts) != 1: 

self.abort('must supply one host') 
if not iface: 

self.abort('must supply iface') 
if not mac: 

self.abort('must supply mac') 


for host in hosts: 
self.db.execute("""update networks, nodes set 
networks .mac=NULLIFC'%s' , 'NULL') where 
nodes.name-'96s' and networks.node-nodes.id and 
(networks .device-'96s' or networks .mac-'96s')'"" 96 
(mac, host, iface, iface)) 
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Help and Docstrings 


4 The command line is the documentation 
> No more out of date man pages 


5 Still needs a cookbook document, but 
reference is part of the code 


€ We've been looking at this all session 
+ Class docstring """text""" 
4 Command line has an XML format 
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# rocks list roll help 
rocks list roll [roll]... 


Description: 


List the status of available rolls. 


Arguments : 


[roll] 


List of rolls. This should be the roll base name (e.g., base, hpc, 
kernel). If no rolls are listed, then status for all the rolls are 
listed. 


Examples: 
$ rocks list roll kernel 
List the status of the kernel roll 
$ rocks list roll 


List the status of all the available rolls 
(€ 2008 UC Regents 
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اقم 


11: 


: import os 


import stat 
import time 
import sys 
import string 


: import rocks.c 


rocks list roll help 


: class Command(rocks . commands . Rol LArgumentProcessor , 


rocks . commands .1ist.comnand): 


List the status of available rolls. 


«arg optional="1" type-" string" name-'roll' repeat-'1'» 

List of rolls. This should be the roll base name (e.g., base, hpc, 
kernel). If no rolls are listed, then status for all the rolls are 
listed. 

«/arg» 


«example amd-'list roll kernel '> 
List the status of the kernel roll 
</example> 


«example amd-'list roll'> 
List the status of all the available rolls 
</example> 


def run(self, params, args): 


self.beginOutput() 
for (roll, version) in self.getRollNames(args, params): 
self.db.execute("""select version, arch, enabled from 
rolls where name-'Xs' and version="%s"""" 96 
(roll, version)) 
for row in self.db.fetchallO: 
self.addOutput(roll, row) 
(€ 2008 UC Regents 108 
self.endOutput(header-['name', 'version', 'arch', 'enabled'], 
trimOwner=2) 





<arg> 


+ Attributes 
5 name (required) 
> optional (default = “O”) 
2 type (default = “string”) 
> repeat (default = “0”) 
+ Example: 


«arg type-" string" name-'network' repeat='1'> 
One or more named networks that should have the 
defined netmask. 
«/arg» 
O 2008 UC Regents 
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<param> 


4 Attributes 
5 name (required) 
> optional (default = "1”) 
2 type (default = "string") 
> repeat (default = “O”) 


+ Example: 
<param type-'string' name-'iface'» 

Can be used in place of the iface argument. 
</param> 
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<example> 


4 Attributes 
> cmd(required) 
€ Example: 


«example cmd='set host interface mac compute-0-0 
eth1 00:0e:0c:a7:5d:ff'» 
Sets the MAC Address for the eth1 device on host 
compute-0-0. 

</example> 
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<related> 


Example 


<related>set host interface 1face</related> 
<related>set host interface 1p</related> 
<related>set host interface gateway</related> 
<related>set host interface module</related> 
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e 


Help 


4 rocks «verb» «object...» «subject» help 
> Loads the command module 
> Parses the XML docstring 
> Format and output help as 80 column text 


+ Debug syntax with format= parameter 
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help format=raw 


rocks list roll help format=raw 
List the status of available rolls. 


# 
1 
2 
3 
4: «arg optional-'1' type='string' name-'roll' repeat='1'> 

5: List of rolls. This should be the roll base name (e.g., base, hpc, 
6: kernel). If no rolls are listed, then status for all the rolls are 
7: listed. 

8: «/arg» 

9 


10: «example cmd-'list roll kernel'> 
11:List the status of the kernel roll 
12: examples 


14: «example cmd-'list roll'> 
15:List the status of all the available rolls 
16: </example> 
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Help format=parsed 


# rocks list roll help format=parsed 


i'related': [], "example": [Cu'list roll kernel", u'\t\t\n\tList the 
status of the kernel roll\n\t'), (u'list roll", u'\n\tList the status 
of all the available rolls\n\t')], "description": u'\n\tList the status 
of available rolls.\n\t\n\t', "param": [], "arg": [CCu'roll', 
u'string', 1, 1), u'NnNtList of rolls. This should be the roll base 
name (e.g., base, hpc,\n\tkernel). If no rolls are listed, then status 
for all the rolls are\n\tListed.\n\t')]} 
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Docbook 


4 Roll Usersguide Command Reference is 
generated automatically 


# rocks list roll help format=docbook 
«section id-"rocks-list-roll" xreflabel="list roll"> 
<title>list roll</title> 
<cmdsynopsis> 

<command>rocks list roll</command> 

«arg rep-"repeat" choice="opt">roll</arg> 
</cmdsynopsis> 
<para> 


List the status of available rolls. 
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<variablelist><title>arguments</title> 
<varlistentrv> 





AVOIDING BECOMING A 
DEVELOPER 


O 2009 UC Regents 
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Philosophy 


4 All software is installed 
on the local disk 
€ Does not require NFS or 
non-scalable diskless 
technologies 
€ Use the native OS 
packager for everything 
5 Linux = rpm 
5 Solaris = pkg 
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Violate the Rules 


4 You just need a few 
packages added and 
cannot find or build 
packages 


€ You want this only on 
your cluster and not on 
several clusters 

4 You still want to avoid 
NFS and benefit from 
Rocks management 
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Get a Directory Tree 


4 Build your software from source and 
install on the frontend 
> configure 
5 make 
> install 


€ Or, just untar a binary bundle 
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CREATE PACKAGE 


rocks create package 
«path» 
<package-name> 


rocks create package 
/ 021 /mx 


mx 
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Done 


# rpm -qip mx- 
Name : 
Version E 


Release B 
PDT 


Install Date: 
Group B 
Size E 
Signature S 
Summary B 
Description : 
The mx extensi 


which enhance 


1.0-1.x86 64.rpm 


mx Relocations: (not relocatable) 

1.0 Vendor: Rocks Clusters 

1 Build Date: Tue 12 May 2009 04:40:00 PM 
(not installed) Build Host: vizagra.rocksclusters.org 
System Environment/Base Source RPM: mx-1.0-1.src.rpm 

17588899 License: University of California 
(none) 


A collection of Python software tools. 


ons for Python are a collection of Python software tools 


Python"s usability in many areas. 
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ADDING YOUR PACKAGE TO 
COMPUTE NODES 


O 2009 UC Regents 
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Step 1: Contribute the RPM 


4 Your distribution looks for packages from Rolls 
and in a contrib area 


+ Copy your RPMS into contrib 


cp mx-1.0-1.x86 64.rpm 


/export/rocks/install/contrib/5.2/ 
x86 64/RPMS 


O 2009 UC Regents 124 





Step 2: Extend XML 


cd /export/rocks/install/site- 
profiles/5.2/nodes/ 


cp skeleton.xml 


extend-compute. xml 


vi extend-compute.xml 
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Add Package Tag 


original 
<kickstart> 
<description> 


Skeleton KML Node 
</description> 


<changelog> 
</changelog> 


<!— 


<package></package> 
--> 


<post> 
</post> 


</kickstart> 


modified (with mx) 
<kickstart> 

<description> 

Skeleton XML Node 


</description> 


<changelog> 
</changelog> 


<package>mx</package> 


<post> 


</post> 


</kickstart> 
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Step 3: Rebuild Distribution 


$ RPM package is already contributes 
4 XML node file is already extended 
4 Now we need to rebuild the dist 


4 Must be done in /export/rocks/install 
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CREATE DISTRO 


cd /export/rocks/install 


rocks create distro 
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otep 4: Re-install 


(repeated material 3 slides) 


€ PXE Boot 
5 Network Boot is first in BIOS boot order 
5 Set Rocks Boot action to install 
> Reboot the host 


+ Otherwise use old rocks commands or 
just hard power cycle the host. 
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SET HOST BOOT 


rocks set host boot 
«host» 


action=<boot-action> 


rocks set host boot 
compute-0-0 
action=install 
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RUN HOST 


rocks run host 
«host» 


«command» 


rocks run host 
compute-0-0 
/sbin/init 6 


O 2009 UC Regents 





e 


ohould | Build a Roll? 


contrib & site-profiles Roll 

€ Fast and Easy € Takes about 1 day 
€ Admin Friendly + Developer Friendly 
4 Difficult to share € Easy to share (.iso) 


e Difficult to backup/restore e Easy to backup/restore 


4 Frontend is your e Frontend is your 
development host development host 
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Break Time 


= || Bsc € 2010 UC Regents 
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